Methods and systems for deriving a score with which item listings are ordered when presented in search results

ABSTRACT

Methods and systems for analyzing, ordering and presenting item listings are described. In one example embodiment, a search query is processed to identify item listings satisfying the search query. Then, for each item listing that satisfies the search query, a ranking score is derived and assigned to the item listing. The ranking score is based in part on a relevance score, a listing quality score and a business rules score (or, adjustment factor). Finally, the item listings are ordered, based on their corresponding ranking score, and presented in order in a search results page.

RELATED APPLICATIONS

This patent application claims the benefit of the filing date of theprovisional patent application with Application Ser. No. 61/167,796,filed on Apr. 8, 2009, entitled, “METHODS AND SYSTEMS FOR PRESENTINGITEM LISTINGS IN A SEARCH RESULTS PAGE”, which is hereby incorporatedherein by reference.

TECHNICAL FIELD

The present disclosure generally relates to data processing techniques.More specifically, the present disclosure relates to methods and systemsfor managing how search results are processed and presented to a user ofa computer-based trading or e-commerce application.

BACKGROUND

Advancements in computer and networking technologies have enabledpersons to conduct commercial and financial transactions “on-line” viacomputer-based applications. This has given rise to a new era ofelectronic commerce (often referred to as e-commerce.) A number ofwell-known retailers have expanded their presence and reach by operatingwebsites that facilitate e-commerce. In addition, many new retailers,which operate exclusively online, have come into existence. The businessmodels utilized by enterprises operating online are almost as varied asthe products and services offered. For instance, some products andservices are offered at fixed prices, while others are offered viavarious auction methods, and still others are offered via a system ofclassified ad listings. Some enterprises specialize in the selling of aspecific type of product (e.g., books) or a specific service (e.g., taxpreparation), while others provide a myriad of categories of items andservices from which to choose. Some enterprises serve only as anintermediary, connecting sellers and buyers, while others sell directlyto consumers.

Despite the many technical advances that have improved the state ofe-commerce, a great number of technical challenges and problems remain.One such problem involves determining how to best present products andservices (e.g., items) that are being offered for sale, so as tomaximize the likelihood that a transaction (e.g., the sale of a productor service) will occur. For instance, when a potential buyer performs asearch for a product or service, it may often be the case that thenumber of item listings that satisfy the potential buyer's query farexceeds the number of item listings that can practically be presented ona search results page. Furthermore, it is well established that thepresentation of an item listing in a search results page—for example,the order or placement of the item listing in a list of listings, thefont, font size, or color of the listing, and so on—can affect whetherpotential buyers select the listing, and ultimately purchase the listedproduct or service.

For enterprises that serve as an intermediary—for example, by connectingbuyers with sellers—it is generally desirable that the presentation ofitem listings occur in a fair manner that strikes a balance between theneeds and desires of the various sellers, the buyers or potentialbuyers, and the enterprise itself. If a preference is given to oneseller, such that the one seller's item listings are consistently beingpresented in the most prominent position(s) on a search results page,other sellers may not participate, which will ultimately have a negativeimpact on the enterprise. Similarly, if item listings are presented inaccordance with an algorithm that is too rigid and that cannot easily bealtered or tweaked, such as a first-listed first-presented algorithm,some sellers may attempt to game the system, again negatively impactingother sellers, the potential buyers' experience, and ultimately theenterprise itself. Furthermore, using a simple and rigid algorithm forpresenting item listings prevents the enterprise from optimizing thepresentation of item listings to improve the overall conversion rate foritem listings. This may lead potential buyers to shop elsewhere, whichultimately will negatively affect the e-commerce enterprise.

DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings, in which:

FIG. 1 is a block diagram of a network environment including anetwork-connected client system and server system, with which anembodiment of the invention might be implemented;

FIG. 2 illustrates an example of the various functional components ormodules comprising the item listing presentation management module,according to an embodiment of the invention;

FIG. 3 illustrates an example of a formula or equation, which is used toderive a ranking score, in some embodiments of the invention;

FIG. 4 illustrates an example of a method, according to an embodiment ofthe invention, for processing a search query and presenting itemlistings in a search results page;

FIG. 5 illustrates an example of a formula or equation, which is used toderive a listing quality score, in some embodiments of the invention;

FIG. 6 illustrates a table showing how the weighting factor used in thelisting quality formula changes the emphasis from a predicted score, toan observed score, as the number of search impressions increases,according to one embodiment of the invention;

FIG. 7 illustrates a graph showing an example of the behavior of alisting quality score for a “good” (i.e., intrinsically high quality)item listing and a “bad” (i.e., intrinsically low quality) item listingover a period of time, as the search impression count for each itemlistings increase, according to an embodiment of the invention;

FIG. 8 illustrates a block diagram showing how, in some embodiments,item listings may be grouped to ensure new listings are displayed in thesearch results page;

FIG. 9 illustrates a method, according to an embodiment of theinvention, for presenting search results based on a listing qualityscore

FIG. 10 illustrates an example of a business rule, which might be usedto demote an item listing, according to an embodiment of the invention;

FIG. 11 illustrates a chart showing the relationship between an itemlisting's rank and the probability that the item will be purchased, aswell as the effect of a promotion or demotion, according to anembodiment of the invention;

FIG. 12 illustrates a method, according to an embodiment of theinvention, for utilizing a business rule to adjust the ranking scoreassigned to an item listing;

FIG. 13 is a block diagram of a machine in the form of a mobile devicewithin which a set of instructions, for causing the machine to performany one or more of the methodologies discussed herein, may be executed.

DETAILED DESCRIPTION

Methods and systems for processing a search, and presenting searchresults, are described. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of the various aspects of different embodimentsof the present invention. It will be evident, however, to one skilled inthe art, that the present invention may be practiced without thesespecific details.

System Architecture

FIG. 1 is a block diagram of a network environment 10 including anetwork-connected client system 12 and server system 14, with which anembodiment of the invention might be implemented. As illustrated in FIG.1, the server system 14 is shown to include an on-line tradingapplication 16. In this example, the online trading application 16 iscomprised of two primary modules—an on-line trading engine module 18,and an item listing presentation management module 20.

In some embodiments, the on-line trading engine module 18 may consist ofa variety of sub-components or modules, which provide some of thefunctions of an on-line trading application 16. As described morecompletely below, each module may be comprised of software instructions,computer hardware components, or a combination of both. To avoidobscuring the invention in unnecessary detail, only a few of the on-linetrading engine functions (germane to the invention) are describedherein. For example, the on-line trading engine module 18 may include anitem listing management module (not shown) that facilitates thereceiving and storing of data representing item attributes, whichcollectively form an item listing. When a user desires to list a singleitem, or multiple items, for sale, the user will provide informationabout the item(s) (e.g., item attributes). Such information may besubmitted via one or more forms of one or more web pages, or via dropdown lists, or similar user interface elements. The item listingmanagement module receives the item attributes and stores the itemattributes together within a database 22 as an item listing 24. In someinstances, the item listings may be stored in an item listing databasetable. As described in greater detail below, the item attributes of eachitem listing are analyzed to determine a ranking score assigned to itemlistings and used in determining the position of item listings when theitem listings are being presented in a search results page.

The on-line trading engine module 18 may also include one or moremodules for receiving and storing historical data that is used tomeasure the likelihood that an item listing will, if presented in asearch results page, result in a transaction being concluded. Forinstance, in some embodiments, data associated with user-initiatedactivities are analyzed and captured for the purpose of predictingfuture user activities. If a user submits a search request includingcertain search terms, and then proceeds to conclude a transaction for aparticular item (e.g., purchase the item), information from the user'sinteraction with the online trading application will be captured andstored for the purpose of predicting future actions by other users. Someof the data used in this capacity is generally referred to as relevancedata 26 because it is used to determine a measure of relevance betweensearch terms used in a search query, and individual item listings. Forinstance, if a potential buyer submits a search request with the searchterms, “mobile phone”, item listings that have certain item attributesare more likely to result in the conclusion of a transaction ifpresented in a search results page in response to the search request.For instance, continuing with the example search terms, “mobile phone”,given the specific search terms used in the search query, item listingsthat have been designated as being in a certain category of items, suchas “Electronics”, or even more specifically, “Mobile Phones”, are morelikely to result in a transaction if presented in a search results pagethan item listings in other categories, for example, such as“Automobiles” or “Jewelry”. Similarly, given the search terms, “mobilephone”, item listings with titles that include the search terms mayprove more likely to result in a transaction than item listings withoutthe search terms in the title. Accordingly, in some embodiments, theon-line trading engine 18 includes one or more modules for receiving andanalyzing historical data to generate what is referred to herein asrelevance data. The relevance data is used to derive a measure of thelikelihood that item listings with certain item attributes will resultin a transaction if displayed in response to certain search terms beingsubmitted in a search request. The relevance data may be derived andused as described in related patent applications, U.S. application Ser.No. 11/679,973, entitled, “DETERMINING RELEVANCY AND DESIRABILITY OFTERMS”, filed on Feb. 28, 2007, and U.S. application Ser. No.11/821,928, entitled, “ECONOMIC OPTIMIZATION FOR PRODUCT SEARCHRELEVANCY”, filed on Jun. 26, 2007, both of which are incorporatedherein by reference.

The on-line trading engine module 18 may also include one or moremodules for receiving and storing data representing, among other things,a measure of a seller's performance of obligations associated withtransactions in which the seller has participated. For instance, in someembodiments, when a transaction is concluded, a buyer may be prompted toprovide feedback information concerning the performance of a seller. Thebuyer may, for example, rate the accuracy of the seller's description ofan item provided in the item listing. For instance, if the item receivedby the buyer is in poor condition, but was described in the item listingas being in good condition, the buyer may provide feedback informationto reflect that the seller's description of the item in the item listingwas inaccurate. As described more fully below, this information may beused in a variety of ways to derive a ranking score for an item listing.For instance, in some cases, the seller feedback information may be usedto determine a ranking score for another item listing of the sameseller. Such information may be stored in a database 22, as indicated inFIG. 1 by the seller quality data with reference number 30.

As illustrated in FIG. 1, the database is also shown to include businessrules data 28. As described in greater detail below, the business rulesdata 28 is managed and used by a business rules management module forthe purpose of promoting and/or demoting item listings that satisfy asearch query. For instance, when determining the order or arrangement ofitem listings for presentation on a search results page, an item listingmay be promoted—presented in a more prominent position—or,demoted—presented in a less prominent position—based on the evaluationof a business rule that is dependent upon certain business rule data 28.In some embodiments, item attributes and seller attributes may be usedin conjunction with business rule data, for the purpose of evaluatingbusiness rules. In some embodiments, the promotion or demotion may beeffected by multiplying a business rules score and the ranking score.Business rules may be used to promote certain business policies and toimpact user's behavior. For instance, a business rule that provides apromotion to item listings that are offering free shipping will likelyhave the effect of persuading sellers to offer free shipping to havetheir item listings appear in the most prominent positions of the searchresults page. Similarly, demoting item listings based on negative sellerfeedback information will typically motivate sellers to perform theirobligations as agreed upon.

Referring again to FIG. 1, the second primary module of the on-linetrading application 16 is an item listing presentation management module20. The item listing presentation management module 20, which isdescribed more completely in connection with the description of FIG. 2,provides the logic used to assign a ranking score (sometimes referred toas a Best Match Score) to item listings that satisfy a search query, andto use the ranking score to determine the order of item listings whenthe item listings are presented in a search results page.

For instance, in some embodiments, a user operates a web browserapplication 32 on a client system 12 to interact with the on-linetrading application residing and executing on the server system 14. Asillustrated by the example user interface with reference number 34, auser may be presented with a search interface 34, with which the usercan specify one or more search terms to be used in a search requestsubmitted to the on-line trading application 16. In some embodiments, inaddition to specifying search terms, users may be able to select certainitem attributes, such as the desired color of an item, the itemcategories that are to be searched, and so on. After receiving andprocessing the search request, the on-line trading application 16communicates a response to the web browser application 32 on the clientsystem 12. For instance, the response is an Internet document or webpage that, when rendered by the browser application 32, displays asearch results page 36 showing several item listings that satisfy theuser's search request. As illustrated in the example search results page36 of FIG. 1, the item listings are arranged or positioned on the searchresults page in an order determined by the item listing presentationmanagement module 20. The item listings are, in some embodiments,presented by a presentation module, which may be a web server or anapplication server.

In general, the item listings are presented in the search results pagein an order based on a ranking score that is assigned to each itemlisting that satisfies the query. In some embodiments, the item listingswill be arranged in a simple list, with the item listing having thehighest ranking score appearing at the top of the list, followed by theitem listing with the next highest ranking score, and so on. In someembodiments, several search results pages may be required to present allitem listings that satisfy the query. Accordingly, only a subset of theset of item listings that satisfy the query may be presented in thefirst page of the search results pages. In some embodiments, the itemlistings may be ordered or arranged in some other manner, based on theirranking scores. For instance, instead of using a simple list, in someembodiments the item listings may be presented one item listing perpage, or, arranged in some manner other than a top-down list.

As described in greater detail below, the ranking score may be based onseveral component scores including, but by no means limited to: arelevance score, representing a measure of the relevance of an itemlisting with respect to search terms provided in the search request; alisting quality score, representing a measure of the likelihood that anitem listing will result in a transaction based at least in part onhistorical data associated with similar item listings; and, a businessrules score, representing a promotion or demotion factor determinedbased on the evaluation of one or more business rules. As used herein, acomponent score is a score that is used in deriving the overall rankingscore for an item listing. However, a component score in one embodimentmay be a ranking score in another embodiment. For instance, in someembodiments, the ranking score may be equivalent to a single componentscore, such as the listing quality score. Similarly, in someembodiments, the ranking score may be equivalent to the business rulesscore.

FIG. 2 illustrates an example of the various functional components ormodules comprising the item listing presentation management module 20,according to an embodiment of the invention. As illustrated in FIG. 2,the item listing presentation management module 20 is comprised of fourprimary modules, including a ranking score assignment module 40, alisting quality management module 42, an intermingler module 44 and abusiness rules management module 46. In some embodiments, each of thefour primary modules receives some input data and derives a componentscore, or otherwise effects the outcome of the overall ranking score(e.g., the Best Match Score) that is assigned to an individual itemlisting. In addition, the ranking score assignment module 40 facilitatesthe actual assignment of the ranking scores to the individual itemlistings, and in some embodiments, arranges or orders the item listingsbased on their assigned ranking scores. In some embodiments, thepresentation module 60, which may be a document or page generator, a webserver, or application server, facilitates that actual presentation ofthe search results page containing the search results.

The ranking score assignment module 40 includes a search engine module48 and a relevance score module 50. The ranking score assignment module40 assigns a ranking score to each item listing that satisfies a searchquery, such that the ranking score can be used to order or arrange itemlistings when the item listings are presented in a search results page.In some embodiments, the ranking score will be determined based on avariety of component scores (e.g., relevance score, listing qualityscore, business rules score), however, it will be appreciated that theranking score might be based on a single component score (e.g., therelevance score alone), or various combinations of the component scores.In some embodiments, after assigning a ranking score to each itemlisting that satisfies a specific user's search query, the ranking scoreassignment module 40 generates a list, or otherwise orders, the itemlistings that satisfy the search query. Alternatively, in someembodiments, a separate ordering module (not shown) might be utilized toorder the item listings.

In some embodiments, the search engine module 48 provides the actualsearch function. For instance, the search engine module 48, in someembodiments, receives and processes a search request to identify theitem listings that satisfy the search request. It will be appreciated bythose skilled in the art that a variety of search techniques might beimplemented to identify item listings that satisfy a search request. Ingeneral, however, the item attributes of item listings are analyzed forthe presence of the user-provided search terms. For instance, in someembodiments, the title and/or user-provided item description aresearched for the presence of search terms included in the search query.In some instances, particularly when a user is performing an advancedsearch, other item attributes such as a user-specified category for theitem listing may be used to identify item listings satisfying the searchquery. The exact methods and algorithms used to execute a search,however, are beyond the scope of the present application.

In an alternative embodiment, the search engine module 48 may representan interface to a search engine implemented as an external component ormodule, for example, as part of the on-line trading engine module 18, oras a separate external module. In such a scenario, the search enginemodule 48 may simply receive the set of item listings that satisfy asearch query.

As illustrated in FIG. 2, the ranking score assignment module 40includes a relevance score module 50, which determines a relevance scorefor each item listing that satisfies a search query, based at least inpart on the search terms used in the search query. In general, therelevance score assigned to an item listing is based on an analysis ofhistorical data and represents a measure of the likelihood (e.g., aprobability) that the item listing will result in a transaction beingconcluded in view of the particular search terms included in the searchrequest. In some embodiments, the search methods and the calculationand/or determination of a relevance score may be achieved withalgorithms and methods consistent with those described in related patentapplications, U.S. application Ser. No. 11/679,973, entitled,“DETERMINING RELEVANCY AND DESIRABILITY OF TERMS”, filed on Feb. 28,2007, and U.S. application Ser. No. 11/821,928, entitled, “ECONOMICOPTIMIZATION FOR PRODUCT SEARCH RELEVANCY”, filed on Jun. 26, 2007.

The listing quality management module 42 determines a listing qualityscore, which, in some embodiments, is used as a component score indetermining the ranking score for each item listing that satisfies aquery. As described in greater detail below, the listing quality scorerepresents a measure of the likelihood that an item listing, ifpresented in a search results page, will result in a transaction beingconcluded. The listing quality score, in some embodiments, is derived asa weighted sum of a first part—a predicted score based on itemattributes of the item listing, and in some cases seller attributes,that are determinable at the time of listing—and a second part—a scorebased on demand metrics that are observed over time. As such, thelisting quality management module is shown in FIG. 2 to include apredicted score module 52 and an observed score module 54 for derivingthe predicted listing quality score, and the observed listing qualityscore.

In some embodiments, the predicted score module 52 determines thepredicted listing quality score (e.g., the first part) by comparing itemattributes of the item listing to which the listing quality score isbeing assigned, with item attributes of item listings that aredetermined to be similar. For instance, if a particular item listing isoffering for sale a television set at a particular price (e.g., $100),the particular price may be compared with the prices of similartelevision sets that have previously been sold. If, for example, theprice of the television set is lower than the mean, mode, or medianprice (e.g., $200) of similar television sets, which have sold recently,then the listing quality score should reflect the “good” price of thetelevision set offered via the item listing.

In some embodiments, the observed score module 54 determines theobserved listing quality score (e.g., the second part) by deriving andanalyzing demand metrics representing the actual historical performanceof the item listing, or item listings determined to be similar. Theactual historical performance may be based on some time-based metric.For instance, in some embodiments, the demand metric may be the numberof transactions (e.g., sales) per search impression, which may generallybe referred to as a conversion rate. Accordingly, if a particular itemlisting results in a transaction being concluded (e.g., a sale) everytime the item listing is presented in a search results page, then thisstrong performance of the item listing should be reflected in thelisting quality score. Similarly, if an item listing has lots of searchimpressions—that is, it has been presented in a search results pageseveral times—but has no, or only a few, transactions, then the itemlisting may not be a strong performing item listing, and this poorperformance should be reflected in a lower listing quality score.

Of course, a single quantity item listing—that is, an item listingoffering only a single item—will not have the benefit of the demandmetric, transactions per search impressions, as the item listing expiresupon the first (and only) transaction (e.g., sale of the item).Accordingly, in some embodiments, for some types of item listings, suchas auctions and single item fixed price listings, the demand metric fordetermining the observed listing quality score may be based on thenumber of views, the number of bids (for an auction), or the number ofwatch lists associated with the item listing. Alternatively, in someembodiments, the demand metric for singe-item listings (both auctionsand fixed price) may be based on the performance of item listingsdetermined to be similar to the single quantity item listing. The demandmetric for a multi-quantity item listing, however, may be based on theperformance of the actual item listing, item listings determined to besimilar, or some combination.

When an item listing is first generated and initially listed, the itemlisting will not have any historical data available for assessing itsactual performance. Therefore, the listing quality score of a new itemlisting is based primarily on a prediction (e.g., the predicted listingquality score), taking into consideration the historical performance ofitem listings with similar item attributes. However, over time, as thenew item listing is presented in search results pages, historical datafor assessing the performance of the item listing will become available.Consequently, in some embodiments, the listing quality score is based ona weighted sum of the predicted score and the observed score, such that,over time, the weight is shifted to move the emphasis from the predictedlisting quality score to the observed listing quality score, ashistorical data becomes available to assess the actual performance ofthe item listing, and thus the demand for the item offered via the itemlisting. For instance, the weighting may be adjusted over time, suchthat, in the beginning, when an item listing is new, the emphasis is onthe predicted score component. This is because the observed data fordetermining the demand metrics may not yet be available. Over time,however, as the item listing is displayed in search results pages, theactual historical data used for deriving the observed demand metric maybe collected. As such, over time, the weight applied to the predictedlisting quality score may be reduced, and the weight applied to theobserved listing quality score (e.g., the demand metric(s)) may beincreased. Accordingly, as shown in FIG. 2, in some embodiments, aweight calculation module 56 determines the weighting factor used inderiving the listing quality score for each item listing.

In some embodiments, item listings may be grouped by some itemattribute, such that item listings from each group are intermingled withone another, according to some predefined ratio, when presented in asearch results page. For example, in some embodiments, item listings mayhave an item attribute that indicates whether the item(s) offered viathe item listing is being offered via an auction process, or via a fixedprice. The intermingler module 44 may ensure that the search resultspage includes item listings from both groups (e.g., auction and fixedprice) in quantities established by some predefined ratio. Similarly,item listings may be grouped into “known” item listings and “unknown”item listings, where “known” item listings represent item listings forwhich historical data is available to derive one or more demand metricsfor the purpose of assessing their actual performance, and “unknown”item listings represent newer item listings for which there isinsufficient historical data available to confidently derive a demandmetric for use in assessing the performance of the item listing.Accordingly, the format mix enforcement module 58 may specify andenforce a certain mix of item listings, based on some item attribute,such as an item attribute indicating the offering is via auction orfixed price, or that an item listing is in the “known” group, or the“unknown” group. Other item attributes may also be used by the formatmix enforcement module 58 and intermingler module 44.

Another module may provide for the general administration, evaluationand enforcement of one or more business rules, as indicated in FIG. 2 bythe business rules management module 46. In some embodiments, businessrules may be used to promote or demote an item listing, such that apromoted or demoted item listing will have its ranking score increased(promotion) or decreased (demotion), effectively repositioning the itemlisting in the search results page to give the item listing a moreprominent position for a promotion, and a less prominent position for ademotion. For example, a business rule may be specified as follows:

IF (CATEGORY=JEWELRY & SELLER QUALITY SCORE<3)=>−20%

In this example, an item listing that has been designated as being inthe category, “JEWELRY”, is to have its ranking score decreased bytwenty percent if the seller quality score is less than three. Thisbusiness rule reflects the desire to ensure that, when a potential buyeris interested in an item of jewelry, the item listings presented in themost prominent positions of the search results page are from sellers whohave good seller quality scores, and thus can generally be trusted.Other business rules may be formulated to promote or demote items basedon a variety of seller attributes and item attributes, including an itemlisting's proposed shipping method and/or cost.

Best Match Scores

In some embodiments, when processing a query resulting from a potentialbuyer's search request, the item listings that satisfy the searchrequest are ordered or otherwise positioned or arranged in a searchresults page (e.g., an Internet document or web page) based on a rankingscore calculated for, and assigned to, each item listing. For instance,in response to receiving and processing a search request, one or morealgorithms are used to assign ranking scores (sometimes referred to as aBest Match Score) to each item listing that satisfies the searchrequest. The ranking scores assigned to the item listings that satisfythe search request are then used to determine where each individual itemlisting is to appear when presented to a user in a search results page.Accordingly, in some embodiments, the item listings that are assignedthe highest ranking scores are placed in the positions deemed to be mostprominent, and therefore most likely to be seen and selected by a user.For example, the item listings with the highest ranking scores may bepositioned at the top of a first page of several search results pagesthat together comprise a long list of item listings that satisfy apotential buyer's search request.

In some embodiments, the ranking score is itself comprised of severalsub-scores or component scores. For instance, as illustrated in FIG. 3,a ranking score 61 (e.g., Best Match Score) may be calculated as theproduct of a relevance score 62, listing quality score 64 and businessrules score 66. Because the ranking score is calculated as a product, ifany component score is low, or zero, it will have a significant impacton the overall score. This is in contrast to a ranking method that usessimple arithmetic to combine component scores. In such a scenario, anitem listing with one low component score may still have a reasonableoverall score if the other component scores are high.

As illustrated in FIG. 3, the ranking score is based in part on arelevance score 62. The relevance score 62 is a measure of the relevanceof an item listing, in light of the search terms submitted by the user.The relevance score may be based on an analysis of an item listing'stitle (e.g., title relevance) as well as an item listing's itemdescription, and historical data indicating how users have previouslyinteracted with item listings having similar item attributes when thoseitem listings were previously presented in a search results page. Therelevance score 62 is calculated at the time the search query isprocessed. Thus, in some embodiments, when the set of item listingssatisfying the query are returned, the item listings are associated witha corresponding relevance score 62, and may be ordered based on theircorresponding relevance scores 62.

As illustrated in FIG. 3, the ranking score is also based in part on alisting quality score 64. The listing quality score 64, which isdescribed in greater detail below in connection with the description ofFIGS. 5, 6, 7 and 8, represents a measure of the quality of the itemlisting. For instance, the listing quality score represents thelikelihood (expressed as a probability in some instances) that an itemlisting will result in conclusion of a transaction, if presented in asearch results page. In some embodiments, the listing quality score iscomputed as a weighted sum of a predicted score—based on item attributesknown at listing time—and, an observed score—based on actual performanceof the item listing, or item listings determined to be similar.

The part of the listing quality score representing the predicted scoreis based on an analysis of item attributes of the item listing, incomparison with item attributes of item listings determined to besimilar. Although many item attributes may be considered in variousembodiments, in some embodiments the price of the item and the shippingcost are the primary predictors of quality. For instance, the price ofan item listing relative to the prices of similar item listings thathave previously resulted in transactions is used as a metric to indicatethe likelihood that an item listing will result in a transaction. If theprice for the item listing is below the median price for similar itemlistings, the likelihood that a transaction will conclude if the itemlisting is presented increases. Similarly, if the price for the itemlisting is greater than the median price for similar item listings, thelikelihood of a transaction concluding decreases. The same generalanalysis can be undertaken for shipping cost as well. In someembodiments, the shipping cost is analyzed separately from the price ofthe item, and in some cases, the shipping cost and price are summed toderive a total price that is used in the analysis.

The listing quality score is also based in part on an observed scorerepresenting a demand metric or combination of demand metrics. A demandmetric represents a measure of the demand for an item based at least inpart on historical data. For instance, in some embodiments, a demandmetric used for calculating a listing quality score is calculated as aratio of the number of transactions concluded per search impressions foran item listing, or for item listings determined to be similar. Forexample, in the case of a multi-quantity item listing—that is, an itemlisting offering multiple items (e.g., one-hundred mobile phones)—theobserved demand metric may be derived as the ratio of the number oftransactions concluded per the number of search impressions for the itemlisting. Again referring to the example item listing for a mobile phone,if five out of ten times the item listing is presented in a searchresults page a buyer concludes a transaction by purchasing a mobilephone, then the transactions per search impressions ratio is fiftypercent (50%). This demand metric may be normalized, such that the finalobserved score takes into consideration the performance of the itemlisting in relation to the performance of other item listings forsimilar items. For instance, for certain categories of items (e.g.,Automobiles, Coins, Stamps, Mobile Phones, and so on), differentobserved scores may be interpreted differently. For instance, ratio ofthe transactions per search impressions with value fifty percent (50%)may be viewed as a “good” ratio, indicating a strong item listingperformance, for one category (e.g., Automobiles), but a “bad” ratio,indicating a weak item listing performance for another category (e.g.,Mobile Phones).

In general, if the ratio of the number of transactions per searchimpressions for an item listing is high, the likelihood that the itemlisting will result in a transaction is also high. However, if the totalnumber of search impressions for a given item listing is low, theconfidence in the demand metric may be low. For instance, if the itemlisting has only one search impression, and that search impressionultimately resulted in a transaction, it may be difficult to predictwhether the item listing is a “good” item listing. Accordingly, and asdescribed more completely below, the weighting factor for the demandmetric may be a function of the number of search impressions for theitem listing, or a metric referred to as time on site (TOS),representing the length or duration of time the item listing has beenactive.

In some embodiments, the weighting factor is a function of a time-basedmetric, such that, when the item listing is first listed, the emphasisis on the predicted score, but over time, the emphasis is shifted to theobserved score. For example, in some embodiments, the weighting factoris a function of the number of search impressions that an item listinghas received. For instance, when the search impression count (i.e., thenumber of times an item listing has been presented in a search resultspage) reaches some minimum threshold, the weighting factor applied tothe predicted score is decreased, resulting in less emphasis on thepredicted score, and the weighting factor applied to the observed metricis increased, resulting in greater emphasis on the observed metriccomponent of the listing quality score.

In some embodiments, a single algorithm is used to assign a score toeach item listing. For example, after determining which item listingssatisfy a search request, a single algorithm is used to assign a scoreto each item listing determined to satisfy the search request. Thisscore (e.g., the Best Match Score) is then used to determine theposition of each item listing when the item listings are presented to auser, for example, in a search results page. Alternatively, in someembodiments, several algorithms are utilized to assign scores to itemlistings. For instance, in some embodiments, a separate algorithm isused to assign scores to item listings based on a characteristic of theitem listings, such that the algorithm used for any particular itemlisting is selected based on a particular characteristic or attributevalue of the item listing. For instance, when the total population ofitem listings includes both auction listings and fixed-price listings,one algorithm may be used to assign a score to all item listings thatinclude a fixed price, while a separate algorithm may be used for itemlistings that are offered at auction. Similarly, with multiple-quantityitem listings offered at a fixed price—that is, item listings that offermultiple units of the same item at a fixed price—separate algorithms maybe used to assign scores to those item listings that are new—and thus donot have available historical data for assessing the quality of the itemlisting—and those item listings that are older—and thus have availablehistorical data for assessing the quality of the item listing.Similarly, different algorithms may be used to assign listing qualityscores to item listings in different categories. For instance, somecategories (e.g., Mobile Phones) may use transactions per searchimpressions as the observed demand metric for the observed listingquality score, whereas item listings in another category (e.g.,Automobiles) may use the ratio of views per search impression as thedemand metric for the observed listing quality score.

FIG. 4 illustrates an example of a method, according to an embodiment ofthe invention, for processing a search query and presenting itemlistings in a search results page. The method begins at operation 70,when a search query is received and processed to identify item listingsthat satisfy the search query. As part of processing the search query, arelevance score is assigned to each item listing satisfying the searchquery. Accordingly, in some embodiments, the set of item listingssatisfying the search query may be received, ordered based on theircorresponding relevance scores.

At method operation 72, for each item listing satisfying the searchquery, a listing quality score is determined. Because the listingquality is not dependent upon the search terms used in the search query,in some embodiments, the listing quality score for each item listing maybe pre-calculated—that is, calculated prior to the search request beingprocessed. Accordingly, in some embodiments, the listing quality scorecan simply be retrieved, for example, by looking up the listing qualityscore in a database. Alternatively, in some embodiments, only a portionof the underlying data used for deriving the listing quality score maybe pre-calculated. Therefore, in some embodiments, the listing qualityscore for each item may be calculated in response to the execution ofthe search query and the identification of the item listings satisfyingthe query.

In any case, at method operation 74, one or more business rules areevaluated for the item listings satisfying the search query. Theevaluation of a business rule will, for some item listings, result in abusiness rules score that has the effect of promoting or demoting anitem listing, for instance, by increasing or decreasing the overallranking score of the item listing. For instance, when the business rulesare evaluated, if the ranking score is equivalent to the relevance scoremultiplied by the listing quality score, upon evaluating the businessrules, the ranking score will be multiplied by a business rules score.In some embodiments, a business rules score of one hundred percent(100%) is equivalent to no change—that is, no demotion, and nopromotion. Likewise, a business rules score less than one hundredpercent will result in a demotion, while a business rules score greaterthan one hundred percent will result in a promotion.

At method operation 76, the relevance score, listing quality score, andbusiness rules score are multiplied together to derive a ranking scorefor each item listing. Finally, at method operation 78, the itemlistings are sorted in accordance with their corresponding rankingscore, and presented in a search results page.

Although not shown in FIG. 4, in some embodiments, an interminglermodule 44 may operate to reorder the item listings to ensure that theitem listings presented on a single search results page include a numberof item listings from two or more categories that is consistent with apredefined ratio. For instance, in some embodiments, the item listingsmay be categorized as being in one of two formats—auction format, orfixed price format. In such a scenario, it may be desirable to have acertain ratio (e.g., one-to-one) of auction format item listings tofixed price format item listings. To ensure that this ratio is met, theintermingler module may reorder the item listings, for example, byadjusting the ranking score of the item listings. In variousembodiments, this may occur at different steps in the processing. Forinstance, in some embodiments, the intermingling may occur after theranking score has been derived by multiplying the relevance score, andthe listing quality score, and the business rules score. In otherembodiments, the intermingling may occur after the product of therelevance score and listing quality score is calculated, but beforeadjusting for promotions or demotions based on evaluating businessrules.

Listing Quality Score

As briefly noted above, in some embodiments, the ranking score (e.g.,the Best Match Score) assigned to an individual item listing is based inpart on a listing quality score representing a measure of the overallquality of an item listing. For instance, the quality of an item listingmay be viewed as the probability that an item being offered for sale viathe item listing will be purchased if the item listing is presented in asearch results page. In general, with all else equal, those itemlistings that satisfy the user's search query and have the highestlisting quality scores are presented in the most prominent positions ofthe search results page. As described in detail below, many factors orsub-components may go into determining the listing quality score for anitem listing. Additionally, item listings may be assigned to differentgroups or “buckets” based on certain characteristics of the itemlistings, such that a different algorithm is used to determine thelisting quality score used in deriving an item listing's ranking score,based on the assigned group or bucket. Similarly, item listings assignedto certain groups may be intermingled (e.g., rearranged or re-ranked) toensure that a certain ratio or mix of item listings, based on theirassigned category, are presented in a search results page.

As illustrated in FIG. 5, the listing quality score 82 may be calculatedwith a formula taking the general form of the equation with referencenumber 80. In this equation 80, “LQ” stands for Listing Quality, and thesymbol “*” represents a multiplication operation. As shown in theequation 80 presented in FIG. 5, the score 82 in some embodiments iscalculated as a weighted sum of two parts—a predicted listing qualityscore 84, based in part on an analysis of item attributes and sellerattributes known at listing time—and, an observed listing quality score86, based on an analysis of an item listing's actual performance overtime. The item attributes and seller attributes used for deriving thepredicted listing quality score 84 are generally static in nature and,with a few exceptions, are not expected to change from the time the itemlisting is first generated. The observed listing quality score 86 isgenerally based on historical data obtained over the life of the itemlisting, and as such, is considered to be based on dynamic data.

Because a new item listing will not yet have been presented in a searchresults page, a new item listing will not have any associated historicaldata by which its performance can be measured. Accordingly, for new itemlistings, the listing quality score 82 is based primarily upon thepredicted listing quality score 84, and is essentially a prediction ofhow the item listing will perform, based on an analysis of the itemattributes, and in some cases seller attributes, of the item listing. Inparticular, the analysis used in determining the predicted listingquality score 84 involves comparing the item attributes of the itemlisting with item attributes of other similar item listings for whichhistorical performance data is available. For example, if the price ofan item is higher or lower than some measure of central tendency (e.g.,median, mean or mode) for the prices of similar items, then thisinformation can be used to predict how the new item listing willperform.

In addition to the price of an item, a variety of other item attributesmay be considered in deriving the predicted listing quality score 84 forthe overall listing quality score 82, including: the condition of theitem, the shipping method and cost, the duration or length of time theitem listing has been active, a seller's prior conversion rate for allitems or items in a particular category, as well as a seller's rating(e.g., based on feedback provided by buyers). In general, a high pricewill result in a lower listing quality score 82—particularly a highprice relative to the mean, median or mode price of similar itemlistings. Similarly, the higher the cost of shipping relative to thecost of shipping for similar item listings, the lower the listingquality score will be. With respect to the condition of the item, thebetter the condition stated by the seller, typically the better thelisting quality score will be. Of course, a great number of other itemor seller attributes might be considered when deriving the predictedlisting quality score 84.

Various methods may be used to identify those item listings deemedsimilar to the item listing to which the listing quality score is beingassigned. For instance, a comparison of the titles may be made, suchthat item listings using one or more of the same key words in theirtitles may be deemed similar. Additionally, two item listings may bedeemed similar only when the item listings have similar titles and theyare assigned to the same category, or are listed on the same website.Additional constraints may also be used. For instance, an item listingmay be deemed similar to the item listing to which the listing qualityscore is being assigned if the prices of the item listings are within acertain percentage of one another. For example, this prevents anunreasonable comparison of a Mercedes Benz toy car priced at $3.00, witha Mercedes Benz classic car priced at $300, 000. In some embodiments,only a certain percentage of the most similar items are used whendetermining the predicted listing quality score. For instance, whendetermining the median price of similar item listings, the median priceof the top 10% of most similar item listings may be determined forcomparison purposes.

In addition, when deriving the predicted listing quality score 84 for anitem listing, the actual performance of item listings determined to besimilar may be considered. Accordingly, the similar item listings thatare of interest are the item listings that have resulted intransactions, or have some historical data available to assess theirperformance. For instance, if an analysis of historical data indicatesthat the conversion rate for a certain product is 90% when the productis priced at or below a particular price, then a prediction can be madethat the likelihood that an item listing for that product with a priceat or below the particular price is 90%. This may result in a highlisting quality score. The number of search impressions, number ofviews, number of watch lists, and/or number of transactions concludedfor item listings determined to be similar may provide an indication ofhow a new item listing will perform. In this case, a search impressionis defined as the presentation of an item listing in a search resultspage. Accordingly, every time an item listing appears in a searchresults page, a search impression counter for the item listing isincreased. A view occurs when an item listing presented in a searchresults page is selected by, and presented to, a user. A watch list is amechanism for monitoring an item listing. For example, a potential buyermay add an item listing to a watch list so that the potential buyerreceives notifications about certain events, such as, the conclusion ofa transaction via an auction, and so forth. The number of unique userswho add an item listing to a watch list can be used as a measure of theinterest (demand) in an item listing. A transaction is the sale of anitem. In some embodiments, when a user purchases an item in quantity, asingle transaction will be counted. For instance, if a user buys fiveitems via a single multi-quantity item listing, the sale of the fiveitems is viewed as a single transaction for the purpose of determining ademand metric used in calculating the predicted score component of alisting quality score.

In some embodiments, a measure of central tendency (e.g., a median, meanor mode) is calculated for a particular item attribute, or demandmetric, for a certain sized subset of similar item listings. Forinstance, for a certain sized subset of the most similar item listings,the median selling price, median shipping cost, median number of searchimpressions, and so forth, are calculated. Then, the item attributes ofthe item listing being assigned the predicted listing quality score arecompared to these median values. If, for example, the median price atwhich transactions were concluded for a set of similar item listings isgreater than the price of an item listing being assigned a listingquality score, then the lower price of the item listing, compared to themedian price of similar item listings, should be reflected in thelisting quality score as a positive (e.g., a higher score). Similarly,if the item listing being assigned a predicted listing quality score hasa shipping cost that is higher than the median shipping cost for the setof similar item listings under consideration, this high shippingcost—relative to the median shipping cost of similar items—is reflectedin the predicted listing quality score as a negative (e.g., lowerscore).

In some embodiments, the predicted listing quality score 84 may be usedas the overall listing quality score 82. However, as shown in theequation 80 of FIG. 5, some embodiments utilize a weighted sum of thepredicted listing quality score 84, and the observed listing qualityscore 86. The demand metrics that may be used to derive the observedlisting quality score 86 are transactions (sales), views, searchimpressions, and watch lists. In some embodiments, one of these demandmetrics may be used, while in other embodiments, some combination of themetrics may be used to form the overall observed listing quality score82. In some embodiments, the exact combination or formula used to derivethe observed listing quality score 86 may differ by category. Forinstance, the observed listing quality score 86 for item listings in the“Automobiles” category may be based on some combination of searchimpressions and watch lists, while the observed listing quality score 82for item listings in the “Jewelry” category may be based on somecombination of watch lists and views.

One of the issues that exists with conventional methods for assessingthe quality of an item listing is that exposure (e.g., searchimpressions) leads to transactions (sales), which leads to moreexposure. Consequently, when a listing quality score depends in part ona demand metric, such as transactions (sales), those item listings thathave performed well in the past tend to be deemed as high quality, andare therefore positioned in the search results page in the mostprominent positions (e.g., at the top of the first page). Of course,being placed in a prominent position on the search results page willnaturally result in additional transactions (sales), leading one tobelieve the item listing is high in quality. This of course makes itdifficult for new item listings, with no historical data available forassessing quality, to break-in to the high quality tier of item listingsand receive placement in the most prominent positions in the searchresults page.

Some embodiments of the invention address this issue in at least twoways. First, as illustrated in FIG. 5, the listing quality score 82 insome embodiments is calculated as a weighted sum of two parts—apredicted listing quality score 84, based on an analysis of itemattributes known at listing time—and, an observed listing quality score86, based on an analysis of an item listing's actual performance overtime. The weighting factor 88 is applied to the two components such thatthe listing quality score 82 for new item listings, which have noattainable performance history, is based primarily on the predictedscore 84. For instance, in some embodiments, when an item listing isfirst listed, the value of the weighting factor, “WEIGHT”, is set tozero. Accordingly, with “WEIGHT” equal to zero, the listing qualityscore 82 is equivalent to the predicted listing quality score 84.However, over time, as actual data are obtained and the performance ofthe item listing can be assessed, the weighting factor is changed,shifting the emphasis to the observed listing quality score 86. Forinstance, with a value for “WEIGHT” of one half (0.5), the weightingfactor for the predicted listing quality score and observed listingquality score is equal—and equivalent to one half.

Referring now to FIG. 6, in some embodiments, the weighting factor—inparticular, the value of the variable expressed in the equation of FIG.5 as “WEIGHT” 84—is derived as a function of a time-based metric. Forexample, the weighting factor may be a function of the length of time anitem listing has been active. Alternatively, the weighting factor may bea function of the search impression count for the item listing. Forinstance, as illustrated in the table of FIG. 6, as indicated by the Xaxis, the weighting factor is a function of the search impression count.When the search impression count for the item listing is relatively low,the amount of historical data available for accurately and confidentlymeasuring the actual performance of the item listing based on observeddemand metrics is still quite low. Consequently, as indicated by the barwith reference number 90, the portion of the listing quality scoreattributable to the observed listing quality score is low—approximately10%. Similarly, the portion of the listing quality score attributable tothe predicted listing quality score is approximately 90%. As the searchimpression count increases, so too does the weighting factor, and thus,the percentage of the overall listing quality score attributable to theobserved demand metric or metrics for the item listing. For example, asillustrated by the bar in FIG. 6 with reference number 92, when thesearch impression count is high, approximately 90% of the listingquality score is attributable to the observed listing quality score,based on historical data indicating the actual performance of the itemlisting.

Referring now to FIG. 7, an example is provided to show the behavior ofa listing quality score for a “good” (i.e., intrinsically high quality)item listing and a “bad” (i.e., intrinsically low quality) item listingover a period of time, as the search impression count for each itemlisting increases. As illustrated in FIG. 7, when the search impressioncount is low, the good item listing represented by the solid line 94 hasa listing quality score that is just above six, while the bad itemlisting represented by the dashed line 96 has a listing quality scorejust below six. As described in connection with the table in FIG. 6,when the search impression count is low, the listing quality score isprimarily based on the predicted listing quality score for both itemlistings. As the search impression count increases, the weighting factorshifts the emphasis from the predicted listing quality score to theobserved listing quality score. As indicated by the terminal point 98 ofthe solid line 94 representing the listing quality score for the gooditem listing, when the search impression count increases, the highvisibility (e.g., increased number of search impressions) of the gooditem listing results in the conclusion of a transaction. However,because the bad item listing does not result in the conclusion of atransaction with the additional search impressions, the listing qualityscore for the bad item listing begins to decrease as the searchimpression count increases. This is in part because the weighting factorfor the listing quality score shifts to emphasize the observed demandmetric (e.g., transactions per search impressions) for the bad itemlisting. Ultimately, as the search impression count becomes high, thelisting quality score tends toward zero.

By basing the listing quality score of an item listing on a weightedcombination of a predicted score and observed demand metrics, new itemlistings are not unduly penalized for their lack of performance data.Moreover, by shifting the emphasis over time from the predicted listingquality score to the observed listing quality score, those item listingsthat perform well are still rewarded over time by achieving higherlisting quality scores, and thus being positioned prominently in thesearch results page.

Another mechanism used to ensure that new item listings are not undulypenalized for their lack of historical data by which their performancecan be measured is the concept intermingling item listings that havebeen assigned to different “buckets” or groups. For example, asillustrated in FIG. 8, the search results satisfying the query are firstassigned a ranking score, and then divided into buckets or groups. Inthis example, the item listings have been divided into three groups:item listings using an auction format 100, fixed price item listingsdetermined to be known 102, and fixed price item listings determined tobe unknown 104. In this case, an unknown item listing is in essence anew, or newer, item listing. In particular, an item listing isdetermined to be unknown when there is not sufficient historical dataavailable on which to confidently derive a meaningful demand metric forthe item listing. For example, an item listing may be categorized in theunknown group until the item listing has been presented in a searchresults page a predetermined number of times, and its search impressioncount reaches some threshold. When the threshold for the searchimpression count for a particular item listing is reached, the itemlisting is re-categorized into the known group. By grouping the itemlistings into known and unknown categories, the intermingler module 106and format enforcement module (not shown) can be used to ensure that newor newer item listings, which do not yet have sufficient data to reachthe top tier of ranking scores, will be presented in the search resultspage. For instance, a ratio may established for the target number ofknown-to-unknown category item listings that are to appear in eachsearch results page. The intermingle module 106, in conjunction with theformat enforcement module, may rearrange the item listings to ensure thedefined ratio is met. Over time, as the search impression count forthese unknown item listing increases, the item listings will “graduate”into the known category.

FIG. 9 illustrates a method, according to an embodiment of theinvention, for presenting search results based on a listing qualityscore. The method begins at method operation 110 when a search query isprocessed to identify item listings that satisfy the search query. Atmethod operation 112, for each item listing that satisfies the searchquery, a listing quality score is assigned to the item listings. Thelisting quality score for each item listing is derived as a weighted sumof two parts—a first part representing a predicted listing qualityscore, based on analysis of item attributes that are known at listingtime—and, a second part representing an observed score based on one ormore demand metrics derived from historical data observed over time. Theweighting factor applied to the two parts is a function of a time-basedmetric, such as the search impression count for the item listings.Accordingly, over time, as the item listing receives search impressions,the emphasis on the overall listing quality score for item listing willshift from the predicted listing quality score to the observed listingquality score. Finally, at method operation 114, the item listings arepresented in a search results page, ordered based at least in part onthe listing quality score assigned to the item listings.

Business Rules

As briefly noted above, in some embodiments, the ranking score (e.g.,the Best Match Score) assigned to each item listing that satisfies auser's search query may be adjusted (up or down) to reflect a promotionor demotion, based on the evaluation of one or more business rules. Forexample, it may be desirable to promote or demote item listings thathave certain item attributes or seller attributes. If, for example, anitem listing has free shipping, the listing may be promoted. Similarly,if a seller has a low seller quality score (based in part on feedbackfrom buyers), an item listing of the seller may be demoted to reflectthe poor seller quality score. In some cases, when the business rulesare transparent and therefore known to both the sellers and buyers,promotions and demotions might be used to encourage certain behavior.

FIG. 10 illustrates an example of a business rule 120, which might beused to demote an item listing, according to an embodiment of theinvention. As illustrated in FIG. 10, the business rule 120 is comprisedof two parts—a conditional statement 122 and an adjustment factor 124.In this example, the conditional statement sets forth two expressionsjoined by an “&” symbol, wherein one expression 126 involves an itemattribute, and the other expression 128 involves a seller attribute.Accordingly, the conditional statement evaluates to true only when bothexpressions are true. In this example, the conditional statement will betrue (and thus satisfied) if the item listing being considered is in thecategory, “JEWELRY”, and the seller quality score for the sellerassociated with the item listing is less than three. If both expressionsare true, the rule is satisfied and the ranking score for the itemlisting is adjusted by the amount specified by the adjustment factor—inthis case, decreased by twenty percent.

In some embodiments, rules can be expressed using Boolean logic, with“AND” and “OR” expressions, and using any number and combination of itemand seller attributes. The business rule 120 expressed in FIG. 10 isbased on the evaluation of both an item attribute (e.g., Category) and aseller attribute (e.g., seller quality score). Although business rulesmay be used to reflect a wide variety of business policies twoparticular examples are worth describing here. First, in someembodiments, business rules may be used to promote item listings thathave free shipping. For instance, when a seller creates an item listingand designates the listing as having free shipping, the item listing mayreceive a promotion, and therefore be displayed in more prominentposition in the search results page. This policy reflects a desire toprevent sellers from generating listings with artificially low priceswith corresponding artificially inflated shipping charges. Similarly, insome embodiments, business rules might be used to demote item listingsthat have a shipping cost that exceeds the median (or, mean or mode)shipping cost for similar items. In a second example, a business rulemay promote or demote an item listing based on the seller's trust scoreor seller quality score. In some embodiments, different business rulesmight be assigned to different categories, different web sites, or basedon different item attributes, such as the listing format (e.g., auction,fixed-price, classified ad, and so on). For instance, with some types ofitems, such as antiques and/or jewelry, the seller trust factor may bemore important, as there may be more fraud prevalent in the trade ofsuch items.

FIG. 11 illustrates a chart showing the relationship between an itemlisting's rank and the probability that the item will be purchased, aswell as the effect of a promotion or demotion, according to anembodiment of the invention. In FIG. 11, the X-axis represents the rankor slot in which an item listing is to be presented. The Y-axisrepresents the relative probability that the item listing will bepurchased. Accordingly, the curved line with reference number 130represents the relative probability that an item will be purchased,given its rank or slot in the search results page. From the chart, itcan be seen that when an item listing is presented in the lowest rankingslot (e.g., slot number one, representing in some embodiments, the topof the list), it has the greatest probability of resulting in atransaction (e.g., a sale). As the slot or rank decreases, so too doesthe probability that the item listing will result in a transaction(e.g., a sale).

Referring again to FIG. 11, it can be seen that the effect of a twentypercent change, resulting from a promotion or demotion, is different,depending on the item listing's current slot position or rank. Forexample, when the item listing's current rank (or, slot position) isnear the top of the search results page, a twenty percentpromotion/demotion has a less significant impact 132 than when the itemlisting's current rank is lower, for example, on the second or thirdpage of the search results page. For instance, the change in rankindicated by the lines with reference number 132 is less significantthan the change in rank indicated by the lines with reference number134.

FIG. 12 illustrates a method, according to an embodiment of theinvention, for utilizing a business rule to adjust the ranking scoreassigned to an item listing. At method operation 140, a conditionalstatement of a business rule is evaluated. The conditional statement maybe in one or several parts, combined by Boolean expressions, and maycall for the evaluation of an expression including an item attribute ora seller attribute. At method operation 142, if the conditionalstatement evaluated in operation 140 is true, then an adjustment factoris applied to a ranking score assigned to the item listing underconsideration. The adjustment factor may be expressed as a percentage bywhich the ranking score is to be increased (for a promotion) ordecreased (for a demotion). Finally, at method operation 144, the itemlisting is presented in a search results page, positioned with the pagerelative to other item listings, based on the adjusted ranking scoreassociated with and assigned to the item listing.

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods described herein may be at least partiallyprocessor-implemented. For example, at least some of the operations of amethod may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The one or more processors may also operate to support performance ofthe relevant operations in a “cloud computing” environment or as a“software as a service” (SaaS). For example, at least some of theoperations may be performed by a group of computers (as examples ofmachines including processors), these operations being accessible via anetwork (e.g., the Internet) and via one or more appropriate interfaces(e.g., Application Program Interfaces (APIs).)

Example Computer System

FIG. 13 is a block diagram of a machine in the form of a mobile devicewithin which a set of instructions, for causing the machine to performany one or more of the methodologies discussed herein, may be executed.In alternative embodiments, the machine operates as a standalone deviceor may be connected (e.g., networked) to other machines. In a networkeddeployment, the machine may operate in the capacity of a server or aclient machine in server-client network environments, or as a peermachine in peer-to-peer (or distributed) network environments. Themachine may be a personal computer (PC), a tablet PC, a set-top box(STB), a Personal Digital Assistant (PDA), a mobile telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting instructions (sequential or otherwise) that specify actions tobe taken by that machine. Further, while only a single machine isillustrated, the term “machine” shall also be taken to include anycollection of machines that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The example computer system 1500 includes a processor 1502 (e.g., acentral processing unit (CPU), a graphics processing unit (GPU) orboth), a main memory 1501 and a static memory 1506, which communicatewith each other via a bus 1508. The computer system 1500 may furtherinclude a display unit 1510, an alphanumeric input device 1517 (e.g., akeyboard), and a user interface (UI) navigation device 1511 (e.g., amouse). In one embodiment, the display, input device and cursor controldevice are a touch screen display. The computer system 1500 mayadditionally include a storage device (e.g., drive unit 1516), a signalgeneration device 1518 (e.g., a speaker), a network interface device1520, and one or more sensors 1521, such as a global positioning systemsensor, compass, accelerometer, or other sensor.

The drive unit 1516 includes a machine-readable medium 1522 on which isstored one or more sets of instructions and data structures (e.g.,software 1523) embodying or utilized by any one or more of themethodologies or functions described herein. The software 1523 may alsoreside, completely or at least partially, within the main memory 1501and/or within the processor 1502 during execution thereof by thecomputer system 1500, the main memory 1501 and the processor 1502 alsoconstituting machine-readable media.

While the machine-readable medium 1522 is illustrated in an exampleembodiment to be a single medium, the term “machine-readable medium” mayinclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more instructions. The term “machine-readable medium” shallalso be taken to include any tangible medium that is capable of storing,encoding or carrying instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresent invention, or that is capable of storing, encoding or carryingdata structures utilized by or associated with such instructions. Theterm “machine-readable medium” shall accordingly be taken to include,but not be limited to, solid-state memories, and optical and magneticmedia. Specific examples of machine-readable media include non-volatilememory, including by way of example semiconductor memory devices, e.g.,EPROM, EEPROM, and flash memory devices; magnetic disks such as internalhard disks and removable disks; magneto-optical disks; and CD-ROM andDVD-ROM disks.

The software 1523 may further be transmitted or received over acommunications network 1526 using a transmission medium via the networkinterface device 1520 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (“LAN”), a wide area network (“WAN”), theInternet, mobile telephone networks, Plain Old Telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi® and WiMax® networks).The term “transmission medium” shall be taken to include any intangiblemedium that is capable of storing, encoding or carrying instructions forexecution by the machine, and includes digital or analog communicationssignals or other intangible medium to facilitate communication of suchsoftware.

Although an embodiment has been described with reference to specificexample embodiments, it will be evident that various modifications andchanges may be made to these embodiments without departing from thebroader spirit and scope of the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense. The accompanying drawings that form a parthereof, show by way of illustration, and not of limitation, specificembodiments in which the subject matter may be practiced. Theembodiments illustrated are described in sufficient detail to enablethose skilled in the art to practice the teachings disclosed herein.Other embodiments may be utilized and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. This Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

1. A computer-implemented method comprising: processing a query toidentify item listings that satisfy the query; assigning a ranking scoreto each item listing that satisfies the query, the ranking score derivedas the product of two or more component scores including a firstcomponent score representing a measure of relevance between search termsin a search query and the item listing, the second component scorerepresenting a measure of quality for the item listing, independent ofthe search query; and presenting the item listings that satisfy thequery in a search results page, the item listings positioned in thesearch results page based on the ranking score assigned to each itemlisting.
 2. The computer-implemented method of claim 1, whereinpresenting the item listings positioned in the search results page basedon the ranking score assigned to each item listing includes generating alist of the item listings ordered based on the ranking score assigned toeach item listing such that the item listing assigned the highestranking score is first in the list and the item listing assigned thelowest ranking score is last in the list.
 3. The computer-implementedmethod of claim 1, wherein the second component score representing ameasure of quality for the item listing is calculated as a probabilitythat an item listing will conclude a transaction if displayed in asearch results page, the probability derived based on an analysis ofhistorical data for item listings determined to be similar.
 4. Thecomputer-implemented method of claim 1, wherein the second componentscore representing a measure of quality for the item listing iscalculated as a probability that an item listing will conclude atransaction if displayed in a search results page, the probabilityderived based on an observed demand metric.
 5. The computer-implementedmethod of claim 4, wherein the observed demand metric is the number oftransactions concluded per the number of search impressions for the itemlisting.
 6. The computer-implemented method of claim 1, wherein itemlistings determined to be similar are item listings in the same categoryas the item listing to which the ranking score is being assigned.
 7. Thecomputer-implemented method of claim 1, wherein the second componentscore representing a measure of quality for the item listing iscalculated as a probability that the item listing will conclude atransaction if displayed in a search results page, the probabilityderived based on an analysis of an observed demand metric representing ameasure of performance for the item listing.
 8. The computer-implementedmethod of claim 7, wherein the measure of performance for the itemlisting is derived by determining the ratio of transactions concludedper the number of search impressions for the item listing.
 9. Thecomputer-implemented method of claim 1, wherein the second componentscore representing a measure of quality for the item listing iscalculated as a probability that an item listing will conclude atransaction if displayed in a search results page, the probabilityderived based on an analysis of observed metrics representing a measureof performance for item listings determined to be similar.
 10. Thecomputer-implemented method of claim 1, further comprising: adjusting,with an intermingler module, an order of item listings to ensure that atarget ratio of item listings in different categories is presented inthe search results page.
 11. The computer-implemented method of claim 1,wherein, in addition to the first and second component scores, theranking score is derived as the product of a third component scorerepresenting a business rules score, the business rules score i) derivedby evaluating one or more business rules, and ii) resulting in apromotion or demotion of an item listing.
 12. The computer-implementedmethod of claim 11, wherein the promotion or demotion of the itemlisting represents a change in the position of the item listing in thesearch results page as a result of multiplying the ranking score of theitem listing by the business rules score for the item listing.
 13. Asystem comprising: a search engine module configured to process a queryto identify item listings that satisfy the query; a ranking scoreassignment module configured to assign a ranking score to each itemlisting that satisfies the query, the ranking score derived as theproduct of two or more component scores including a first componentscore representing a measure of relevance between search terms in asearch query and the item listing, the second component scorerepresenting a measure of quality for the item listing, independent ofthe search query; and a presentation module to present the item listingsthat satisfy the query in a search results page, the item listingspositioned in the search results page based on the ranking scoreassigned to each item listing.
 14. The system of claim 13, whereinpresenting the item listings positioned in the search results page basedon the ranking score assigned to each item listing includes generating alist of the item listings ordered based on the ranking score assigned toeach item listing such that the item listing assigned the highestranking score is first in the list and the item listing assigned thelowest ranking score is last in the list.
 15. The system of claim 13,wherein the second component score representing a measure of quality forthe item listing is calculated as a probability that an item listingwill conclude a transaction if displayed in a search results page, theprobability derived based on an analysis of historical data for itemlistings determined to be similar.
 16. The system of claim 13, whereinthe second component score representing a measure of quality for theitem listing is calculated as a probability that an item listing willconclude a transaction if displayed in a search results page, theprobability derived based on an observed demand metric.
 17. The systemof claim 16, wherein the observed demand metric is the number oftransactions concluded per the number of search impressions for the itemlisting.
 18. The system of claim 13, wherein item listings determined tobe similar are item listings in the same category as the item listing towhich the ranking score is being assigned.
 19. The system of claim 13,wherein the second component score representing a measure of quality forthe item listing is calculated as a probability that the item listingwill conclude a transaction if displayed in a search results page, theprobability derived based on an analysis of observed metricsrepresenting a measure of performance for the item listing.
 20. Thesystem of claim 19, wherein the measure of performance for the itemlisting is derived by determining the ratio of transactions concludedper the number of search impressions for the item listing.
 21. Thesystem of claim 13, wherein the second component score representing ameasure of quality for the item listing is calculated as a probabilitythat an item listing will conclude a transaction if displayed in asearch results page, the probability derived based on an analysis ofobserved metrics representing a measure of performance for item listingsdetermined to be similar.
 22. The system of claim 13, furthercomprising: an intermingler module to adjust an order of item listingsto ensure that a target ratio of item listings in different categoriesis presented in the search results page.
 23. The system of claim 13,wherein, in addition to the first and second component scores, theranking score is derived as the product of a third component scorerepresenting a business rules score, the business rules score i) derivedby evaluating one or more business rules, and ii) resulting in apromotion or demotion of an item listing.
 24. The computer-implementedmethod of claim 23, wherein the promotion or demotion of the itemlisting represents a change in the position of the item listing in thesearch results page as a result of multiplying the ranking score of theitem listing by the business rules score for the item listing.
 25. Amethod comprising: presenting item listings that satisfy a search queryin a search results page, the item listings ordered based on a rankingscore assigned to each item listing, the ranking score derived as theproduct of three component scores including a first component scorerepresenting a measure of relevance between search terms in a searchrequest and the individual item listings satisfying the search query, asecond component score representing a measure of the item listingslikelihood to conclude a transaction, if presented in a search resultspage, based on a comparison item attributes of the item listings incomparison with item attributes of item listings determined to besimilar and for which historical data is available for predictingperformance, the third component score representing an adjustment factordetermined based on evaluation of one or more business rules.